Method and system for eye tracking using speckle patterns

ABSTRACT

A method of tracking an eye gaze include obtaining a pre-calibrated speckle map of the eye of the user, determining a starting position of the eye using the pre-calibrated speckle map, and tracking movement of the eye relative to the starting position by: directing a light beam at the eye, detecting a plurality of speckle patterns formed at a detector by a portion of the light beam reflected by the eye, and tracking movement of the eye relative to the starting position by tracking the plurality of speckle patterns from frame to frame.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a divisional application of U.S. patent applicationSer. No. 15/805,635, filed on Nov. 7, 2017, U.S. Pat. No. 10,948,981,issued on Mar. 16, 2021, entitled “METHOD AND SYSTEM FOR EYE TRACKINGUSING SPECKLE PATTERNS,” which claims the benefit of U.S. ProvisionalPatent Application No. 62/420,292, filed on Nov. 10, 2016, entitled“METHOD AND SYSTEM FOR EYE TRACKING USING SPECKLE PATTERNS,” thecontents of which are incorporated by reference in their entireties.

BACKGROUND OF THE INVENTION

Eye-gaze tracking technology can be applied to many fields. For example,eye-gaze tracking may be useful for building a virtual reality (VR) oraugmented reality (AR) headset for providing accurate three-dimensionalrenderings to a user. Other applications may include human-computerinterfaces for assisting disabled people, activity recognition, imageand video compression, computer vision, cognitive studies in medicalresearch, laser refractive surgery, vehicle simulators for in-vehicleresearch, training simulators including sports training simulators,fatigue detection, and the like. Despite the progress made in thedevelopment of eye-gaze tracking technology, there is a need in the artfor improved eye-gaze tracking methods.

SUMMARY OF THE INVENTION

The present invention relates generally to methods and systems foreye-gaze tracking. More specifically, the present invention relates tomethods and systems for eye-gaze tracking using speckle patterns.

According to an embodiment of the present invention, a method oftracking movement of an eye of a user includes directing a light beam atthe eye. The eye may reflect a portion of the light beam. The methodfurther includes detecting a plurality of speckle patterns formed at adetector by the portion of the light beam reflected by the eye. Theplurality of speckle patterns may be detected at a predetermined framerate. The method further includes tracking movement of the eye bytracking the plurality of speckle patterns from frame to frame.

According to another embodiment of the present invention, a method oftracking a gaze of an eye of a user includes recording a first pluralityof images of the eye using a camera at a first frame rate, anddetermining a starting position of the eye based on the first pluralityof images of the eye. The method further includes directing a light beamat the eye. The eye may reflect a portion of the light beam. The methodfurther includes detecting a first plurality of speckle patterns formedat a detector by the portion of the light beam reflected by the eye. Theplurality of speckle patterns may be detected at a second frame rategreater than the first frame rate. The method further includes trackingmovement of the eye relative to the starting position by tracking thefirst plurality of speckle patterns from frame to frame.

According to yet another embodiment of the present invention, a methodof tracking a gaze of an eye of a user includes obtaining apre-calibrated speckle map of the eye of the user, determining astarting position of the eye using the pre-calibrated speckle map, andtracking movement of the eye relative to the starting position. In oneembodiment, obtaining a pre-calibrated speckle map of the eye of theuser is performed by: directing a first light beam at the eye, detectinga plurality of first speckle patterns formed at a detector by theportion of the first light beam reflected by the eye, each of theplurality of first speckle patterns corresponding to a respective firstposition of the eye, recording a plurality of images of the eye using acamera, each of the plurality of images of the eye being recorded when arespective first speckle pattern is detected, determining the respectivefirst position of the eye corresponding to each respective first specklepattern based on the respective image of the eye, and storing theplurality of first speckle patterns and the corresponding firstpositions of the eye. In one embodiment, determining a starting positionof the eye using the pre-calibrated speckle map is performed by:directing a second light beam at the eye, the eye reflecting a portionof the second light beam, detecting a plurality of second specklepatterns formed at the detector by the portion of the second light beamreflected by the eye, and determining the starting position of the eyebased on comparisons between the plurality of second speckle patternsand the stored plurality of first speckle patterns and the storedcorresponding first positions of the eye. In one embodiment, trackingmovement of the eye relative to the starting position is performed by:directing a third light beam at the eye, the eye reflecting a portion ofthe third light beam, detecting a plurality of third speckle patternsformed at the detector by the portion of the third light beam reflectedby the eye, the plurality of third speckle patterns being detected at apredetermined frame rate, and tracking movement of the eye relative tothe starting position by tracking the third plurality of specklepatterns from frame to frame.

In a further embodiment of the present invention, a method ofidentification of a user includes obtaining a pre-calibrated speckle mapof a first eye of a first user by: directing a first light beam at thefirst eye, the first eye reflecting a portion of the first light beam,detecting a plurality of first speckle patterns formed at a detector bythe portion of the first light beam reflected by the first eye, each ofthe plurality of first speckle patterns corresponding to a respectivefirst position of the first eye, recording a plurality of images of thefirst eye using a camera, each of the plurality of images of the firsteye being recorded when a respective first speckle pattern is detected,determining the respective first position of the first eye correspondingto each respective first speckle pattern based on the respective imageof the first eye, and storing the plurality of first speckle patternsand the corresponding first positions of the first eye. The methodfurther includes directing a second light beam at an eye of a user, theeye reflecting a portion of the second light beam, detecting a pluralityof second speckle patterns formed at the detector by the portion of thesecond light beam reflected by the eye, each of the plurality of secondspeckle patterns corresponding to a respective second position of theeye, and determining that the user is the first user by comparing theplurality of second speckle patterns with the plurality of first specklepatterns. In some embodiments, each of the portion of the first lightbeam reflected by the eye and the portion of the second light beamreflected by the eye is diffusely reflected or specular reflected by theeye.

Numerous benefits are achieved by way of the present invention overconventional techniques. For example, embodiments of the presentinvention can provide methods and systems for accurate detection ofsmall eye-gaze movements, for example at a micron scale or less, bytracking the movements of speckle patterns. Eye-gaze tracking usingspeckles can be very robust and the tracking quality can be relativelyinsensitive to sensor location with respect to the eye. The methods ofeye-gaze tracking according to embodiments of the present invention mayafford lower power consumption as compared to conventional methods,because of less amount of computation and less number of light sourcesare required (e.g., only one light is required compared to four LEDstypically needed in camera-based eye-tracking). The methods may alsoafford lower cost as they only require one light source and onedetector, and do not require a camera lens.

These and other embodiments of the invention, along with many of itsadvantages and features, are described in more detail in conjunctionwith the text below and attached figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows four images (a)-(d) of an eye captured when the eye gazesat various positions.

FIG. 2 illustrates schematically eye movements along three independentaxes.

FIG. 3 illustrates an exemplary speckle pattern generated from a laserpointer.

FIG. 4 illustrates schematically how a speckle pattern may be formed onan image sensor.

FIGS. 5A and 5B show two snapshots of a speckle pattern taken at a smalltime interval with respect to each other.

FIG. 6 illustrates schematically an exemplary setup for eye gazetracking according to an embodiment of the present invention.

FIG. 7A shows a front view of an eye captured by a camera according toan embodiment of the present invention.

FIG. 7B shows a speckle pattern captured by a detector according to anembodiment of the present invention.

FIG. 8 shows a schematic diagram illustrating how a light source, adetector, and an optional eye camera may be mounted on a pair of eyeglasses that can be worn by a user, according to an embodiment of thepresent invention.

FIGS. 9A-9F show six sequential frames of speckle patterns of an eye,captured by the detector at 10 ms (i.e., 0.01 sec) time intervals as theeye moves from one side to another.

FIGS. 10A-10C show three consecutive frames of the detector captured at10 ms intervals illustrating how a speckle pattern may be formed on adetector.

FIG. 11A shows a snapshot of a front view of an eye captured by an eyecamera, according to an embodiment of the present invention.

FIG. 11B shows a snapshot of a speckle pattern generated of the eyecaptured by the detector at the same time as the snapshot of the frontview of the eye shown in FIG. 10A was captured, according to anembodiment of the present invention.

FIG. 11C shows the trajectory of the eye movement from a startingposition to an end position, according to an embodiment of the presentinvention.

FIG. 12A shows snapshots of the speckle patterns at two differentdetectors (two left panels), and snapshots of a front view and a sideview of the eye (two right panels) captured by two different cameras,according to an embodiment of the present invention.

FIGS. 12B and 12C show the trajectories of the eye detected by the twodetectors, respectively, according to an embodiment of the presentinvention.

FIG. 13A shows a snapshot of a front view of an eye when it blinkedaccording to an embodiment of the present invention.

FIG. 13B shows the image at the detector when the eye blinked accordingto an embodiment of the present invention.

FIG. 14A shows: (i) in the two right panels, snapshots of a front viewand a side view of the eye captured by two different eye cameras at thesame time; and (ii) in the two left panels, snapshots of the specklepatterns captured by two different detectors, according to an embodimentof the present invention.

FIG. 14B shows a part of a trajectory of the eye movement according toan embodiment of the present invention.

FIG. 14C shows a tracking quality score as a function of time accordingto an embodiment of the present invention.

FIGS. 15A-15C are similar to FIGS. 14A-14C but are snapshots of adifferent time, according to an embodiment of the present invention.

FIG. 16 illustrates a simplified flowchart illustrating a method ofeye-gaze tracking according to an embodiment of the present invention.

FIG. 17 illustrates a simplified flowchart illustrating a method ofeye-gaze tracking according to another embodiment of the presentinvention.

FIG. 18 is a simplified flowchart illustrating a method of trackingmovement of an eye of a user according to an embodiment of the presentinvention.

FIG. 19 is a simplified flowchart illustrating a method of tracking agaze of an eye of a user according to an embodiment of the presentinvention.

FIG. 20 is a simplified flowchart illustrating a method of tracking agaze of an eye of a user according to another embodiment of the presentinvention.

FIG. 21 is a simplified flowchart illustrating a method of obtaining apre-calibrated speckle map of the eye of the user according to anembodiment of the present invention.

FIG. 22 is a simplified flowchart illustrating a method of determining astarting position of the eye using the pre-calibrated speckle mapaccording to an embodiment of the present invention.

FIG. 23 is a simplified flowchart illustrating a method of trackingmovement of the eye relative to the starting position according to anembodiment of the present invention.

FIG. 24 is a simplified flowchart illustrating a method ofidentification of a user according to an embodiment of the presentinvention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

The present invention relates generally to methods and systems foreye-gaze tracking. More particularly, the present invention relates tomethods and systems for eye-gaze tracking using speckle patterns. Themost widely used current designs are video-based eye trackers, alsoreferred to as passive tracking. A camera focuses on one or both eyesand records their movement as the viewer looks at some kind of stimulus.FIG. 1 shows four images (a)-(d) of an eye captured when the eye gazesat various positions. By tracking the (x, y) coordinates of the centerof the pupil, the eye gaze may be monitored.

Most modern eye-trackers use the center of the pupil and infrared suchas near-infrared non-collimated light to create corneal reflections(CR). The vector between the pupil center and the corneal reflectionscan be used to compute the point of regard on surface or the gazedirection. Two general types of infrared or near-infrared (also known asactive light) eye tracking techniques are used: bright-pupil anddark-pupil. Their difference is based on the location of theillumination source with respect to the optics. If the illumination iscoaxial with the optical path, then the eye acts as a retroreflector asthe light reflects off of the retina creating a bright pupil effectsimilar to red eye. If the illumination source is offset from theoptical path, then the pupil appears dark because the reflection fromthe retina is directed away from the camera.

FIG. 2 illustrates schematically possible eye movements along threeindependent axes: horizontal displacement (first dimension), verticaldisplacement (second dimension), horizontal rotation (third dimension),vertical rotation (fourth dimension), cyclotorsion (fifth dimension),and axial displacement (sixth dimension).

Embodiments of the present invention provide methods and systems fortracking eye gaze using speckle patterns. A speckle pattern is anintensity pattern produced by the mutual interference of a set ofwavefronts. Speckle patterns typically occur in diffuse reflections ofmonochromatic light such as laser light. The speckle effect is a resultof the interference of many waves of the same frequency, havingdifferent phases and amplitudes, which add together to give a resultantwave whose amplitude, and therefore intensity, varies randomly. FIG. 3illustrates an exemplary speckle pattern generated from a green laserpointer.

FIG. 4 illustrates schematically how a speckle pattern may be formed onan image sensor 430. When a surface 420 is illuminated by a coherentlight source 410 such as a laser, according to diffraction theory, eachpoint on an illuminated surface acts as a source of secondary sphericalwaves. The light at any point in the scattered light field is made up ofwaves which have been scattered from each point on the illuminatedsurface 420. If the surface 420 is rough enough to create path-lengthdifferences exceeding one wavelength, giving rise to phase changesgreater than 2π, the amplitude, and hence the intensity, of theresultant light varies randomly. For example, as illustrated in FIG. 4,the scattered light waves may interfere out of phase with respect toeach other and result in a dark speckle 442 at certain positions (e.g.,at point A) on the image sensor 430. Likewise, the scattered light wavesmay interfere in phase with respect to each other and result in a brightspeckle 444 at certain positions (e.g., at point B) on the image sensor430. Thus, a speckle pattern may be viewed as a hologram generated by arough surface.

When a rough surface which is illuminated by a coherent light (e.g. alaser beam) is imaged, a speckle pattern is observed in the image plane;this is called a “subjective speckle pattern.” It is called “subjective”because the detailed structure of the speckle pattern depends on theviewing system parameters. For instance, if the size of the lensaperture changes, the size of the speckles change. When coherent lightwhich has been scattered off a rough surface falls on another surface,it forms an “objective speckle pattern.” If a photographic plate oranother 2-D optical sensor is located within the scattered light fieldwithout a lens, an objective speckle pattern is obtained whosecharacteristics depend on the geometry of the system and the wavelengthof the coherent light source.

Embodiments of the present invention include methods and systems fortracking movement of an eye by directing a light beam at the eye andtracking movements of speckle patterns formed by diffuse and specularreflections of the light beam by the eye. Using speckle patterns totrack eye gaze may afford several advantages. While the texture of thesurface of an eyeball may be too smooth to be detected by other means,it can generate observable speckle patterns. Speckle patterns would moverelative to a sensor when the eyeball is moved, as the speckle patternsare generated by the eyeball and are therefore fixed to the eyeball. Bytracking the movements of speckle patterns, it may be possible toaccurately detect small movements of the eyeball, for example at amicron scale or less.

FIGS. 5A and 5B show two snapshots of a speckle pattern on a detectorcaptured at a small time interval with respect to each other, where thetarget moved by a small distance during the time interval. As can beseen, the speckle pattern has moved with respect to the detector. Forexample, the bright spot at position A in FIG. 5A is moved toward theleft to position A′ near an edge of the detector in FIG. 5B. Similarlythe bright spots at position B and C in FIG. 5A are also moved towardthe left to positions B′ and C′, respectively, in FIG. 5B. In order totrack the movements of a speckle pattern, it may be desired that thebeam spot movement on the target between frames is much less than thebeam spot size, and that the speckle movement on the detector betweenframes is much less than the detector size. For example, in someembodiments, a beam spot size ranging from about 5 mm to about 10 mm maybe utilized. As the eye moves between frames, the position of the beamspot on the eye moves accordingly. For example, for a frame rate of 1000frames per second (fps) given a beam spot size of 5 mm, movement of theeye at a rate of about 5,000 mm per second will be within the beam spotsize. Similarly, the speckle pattern on a detector also movesaccordingly as the eye moves between frames. Thus, it may beadvantageous to use a fast detector, for example, a detector operatingat a rate of 1000 fps or faster, so that the above requirements aresatisfied.

FIG. 6 illustrates schematically an exemplary setup for eye-gazetracking according to an embodiment of the present invention. A lightsource 610 may shine a light beam 620 at an eye at a slanted angle. Theeye may reflect a portion of the light beam 620. A detector 630 may bepositioned adjacent the eye to detect speckle patterns formed by theportion of the light beam reflected by the eye. FIG. 7A shows a frontview of the eye as captured by a camera. The bright spot 702 is the beamspot on the eye. FIG. 7B shows a speckle pattern as captured by thedetector 620. As can be seen in FIG. 7B, interference fringes created byspecular reflections are clearly visible in the speckle pattern. In someembodiments, the light source 610 may be configured to emit light beamsin the infrared wavelength range. In some embodiments, the light source610 may be configured to emit light beams in the near infraredwavelength range, e.g., 0.75-2 μm. In some embodiments, the light source610 may comprise a coherent light source such as a laser source, or apartially coherent light source such as light emitting diodes (LEDs), sothat speckle patterns may be detected.

FIG. 8 shows a schematic diagram illustrating how a light source 810, adetector 830, and an optional eye camera 840 may be mounted on a pair ofeye glasses 850 that can be worn by a user, according to an embodimentof the present invention. The light source 810 may shine a light beam820 at an eye of the user. Diffuse and specular reflection of the lightbeam 820 by the eye may form speckle patterns at the detector 830. Insome embodiments, the light source 810 may comprise a coherent lightsource such as a laser. The eye camera 840 may capture images of theeye, from which initial eye gaze positions may be determined, asdiscussed in more detail below.

FIGS. 9A-9F show six sequential frames of speckle patterns generated bythe reflection of a light beam by an eye, captured by a detector at 10ms (i.e., 0.01 sec) time intervals as the eye moves from one side to theother. As can be seen, the speckle patterns move across the detectorarea toward the left from FIG. 9A to FIG. 9F. For example, the brightregion A in FIG. 9A gradually moves toward the left in FIGS. 9B and 9C,and then moves out of the detector area in FIGS. 9D-9F. Similarly, thedark region B in FIG. 9B just starts to appear in FIG. 9B and movesgradually toward the left in FIGS. 9C-9E, and then moves out of thedetector area in FIG. 9F.

FIGS. 10A-10C show three consecutive frames of the detector captured at10 ms intervals illustrating how speckle patterns may be formed on animage sensor. Here, although no interference fringes are visible,speckle patterns created by diffuse reflections of the eye are visible.

By tracking the movements of the speckle patterns, the movement of theeye may be tracked. According to some embodiments of the presentinvention, the movements of the speckle patterns may be tracked using anoptical flow algorithm. Optical flow or optic flow is the pattern ofapparent motion of objects, surfaces, and edges in a visual scene causedby the relative motion between an observer (an eye or a camera) and thescene. Sequences of ordered images may allow the estimation of motion aseither instantaneous image velocities or discrete image displacements.Methods of determining optical flow may include phase correlationmethods, block-based methods, differential methods, the Horn-Schunckmethod, the Buxton-Buxton method, the Black-Jepson method, and generalvariational methods, discrete optimization methods, and the like.According to an embodiment of the present invention, movements of theeye is tracked by applying a phase-correlation algorithm of optical flowto sequential frames of speckle patterns of the eye.

FIGS. 11A-11C illustrate some exemplary results of eye-gaze trackingaccording to an embodiment of the present invention. FIG. 11A shows asnapshot of video of a front view of the eye captured by an eye camera.FIG. 11B shows a snapshot of the speckle pattern of the eye captured bya detector at the same time as the snapshot of the front view of the eyewas captured. FIG. 11C shows the trajectory of the eye movement from astarting point up to the end point where the snapshot of FIGS. 11A and11B are obtained. The horizontal axis and vertical axis in FIG. 11C arehorizontal position and vertical position, respectively, of the eye inarbitrary units (e.g., in terms of pixels of the detector). Aftercalibration, the eye-gaze positions may be translated into gaze angles.The frame rate for detecting the speckle patterns is about 100 framesper second. The duration of the tracking is about 20 seconds. As can beseen in FIG. 11C, the movement of the eye is continuously tracked as theeye is moved back and forth to the right and to the left, and up anddown.

To show the robustness of the eye-gaze tracking using speckle patterns,two detectors may be positioned near the eye to track the eye'smovements. FIG. 12A shows snapshots of the speckle patterns captured bytwo different detectors (two left panels), and snapshots of a front viewand a side view of the eye (two right panels) captured by two differentcameras. FIGS. 12B and 12C show the trajectories of the eye detected bythe two detectors, respectively. The horizontal axis and vertical axisin FIGS. 12B and 12C are horizontal position and vertical position,respectively, of the eye in arbitrary units. The frame rate fordetecting the speckle patterns is about 100 frames per second for eachof the two detectors. The duration of the tracking is about 20 seconds.As can be seen, the two trajectories are consistent with each other,demonstrating that this method of eye-gaze tracking is fairly robust.

According to embodiments of the present invention, movements of the eyemay be continuously tracked by tracking the movements of the specklepatterns of the eye, as long as the eye does not blink. When the eyeblinks, speckle patterns may disappear, and as a result, correlationbetween one frame to the next may be lost. FIG. 13A shows a snapshot ofa front view of an eye when the eye blinked. FIG. 13B shows the image atthe detector captured at the same time when the eye blinked. As can beseen in FIG. 13B, the speckle patterns are lost when the eye blinked. Ifthe blinking lasted long enough, for example a few tens of milliseconds,the speckle patterns captured before and after the blinking may nolonger have phase correlations with each other. As a result, eye-gazetracking may be interrupted, as some algorithms such as the optical flowalgorithms used for eye-gaze tracking according to some embodiments mayrequire phase correlation between sequential frames.

According to some embodiments, a tracking quality score may be used asan indication of the quality of eye-gaze tracking. For example, thetracking quality score may be a measure of a degree of correlationbetween sequential frames of the speckle patterns. In some embodiments,the tracking quality score may be calculated from the value of acorrelation peak. If the correlation between sequential frames isperfect, the value of the correlation peak may be close to unity. If thecorrelation between sequential frames is poor, perhaps due to blinking,the value of the correlation peak may be significantly lower than unity.

FIGS. 14A-14C and 15A-15C illustrate some exemplary results of eye-gazetracking according to an embodiment of the present invention. The tworight panels in FIG. 14A show snapshots of a side view and a front view,respectively, of the eye captured by two different eye cameras. The twoleft panels of FIG. 14A show snapshots of the speckle patterns capturedby two different detectors at the same time the snapshots of the sideview and the front view of the eye were captured. FIG. 14B shows apartial trajectory of the eye movement. The horizontal axis and verticalaxis in FIG. 14B are horizontal position and vertical position,respectively, of the eye in arbitrary units. FIG. 14C shows the trackingquality score as a function of time. The vertical axis is the trackingquality score in arbitrary units. The horizontal axis is frame number(at 10 ms intervals corresponding to 100 frames per second). Asillustrated, the tracking quality score is relatively high for the timeperiod shown. FIGS. 15A-15C are similar to FIGS. 14A-14C but aresnapshots of a different time period. As shown in FIG. 15A, the specklepattern is lost due to blinking of the eye. FIG. 15C shows that thetracking quality score dropped abruptly at about 1060 ms, the momentwhen the eye started to blink.

As illustrated above, accurate tracking of relative motions of the eyecan be achieved by tracking the speckle patterns. But speckle patternsdo not provide absolute eye-gaze position of the eye. According to anembodiment of the present invention, an initial absolute eye-gazeposition of the eye may be determined by another method such as acamera-based method. After the initial eye-gaze position is determined,speckle tracking is used for more accurate tracking of the movements ofthe eye relative to the initial eye-gaze position. Similarly, whencorrelation is lost when the eye blinked, a new initial absoluteeye-gaze position may be determined by a camera-based method. After thenew initial eye-gaze position is established, speckle tracking may beresumed.

FIG. 16 illustrates a simplified flowchart illustrating a method 1600 ofeye-gaze tracking according to some embodiments of the presentinvention. The method 1600 includes performing eye-gaze tracking using acamera (1602), and determining if a starting eye-gaze position has beenobtained (1604). Eye-gaze tracking using a camera may be performed byrecording a plurality of images of the eye using the camera. One of theplurality of images may be selected for determining a starting eye-gazeposition. For example, some images may have relatively poor qualityperhaps because the eye is blinking. Based on the selected image, astarting eye-gaze position may be determined using a pre-calibratedcoordinate system. For example, as illustrated in FIG. 1, the (x, y)coordinates of the center of the pupil may be determined relative to apre-calibrated origin.

If a starting eye-gaze position has not been obtained, the method 1600may continue eye-gaze tracking using the camera (1602) until a startingeye-gaze position has been obtained. Once a starting eye-gaze positionhas been obtained, the method 1600 may start performing eye-gazetracking using speckle patterns (1606). As discussed above, eye-gazetracking using speckle patterns may be performed by directing a lightbeam at the eye, and detecting speckle patterns formed at a detector bya portion of the light beam reflected off of the eye. The light beam maybe produced by a coherent light source such as a laser, or by apartially coherent light source such as light emitting diodes (LEDs), sothat speckle patterns may be detected. Movement of the eye may betracked by tracking the speckle patterns from frame to frame usingalgorithms such as a phase correlation algorithm of an optical flowmethod.

If the user blinks, the ability to track the eye movement using specklepatterns may be lost due to loss of correlation between sequentialframes. Accordingly, the method 1600 further includes determiningwhether blinking has occurred and whether correlation has been lost as aresult of the blinking (1608). In some embodiments, determining whethercorrelation has been lost may include calculating the value of acorrelation peak (as discussed above), and comparing the value of thecorrelation peak to a pre-determined threshold value. If the value ofthe correlation peak is equal or higher than the pre-determinedthreshold value, it may be determined that correlation has not beenlost. Conversely, if the value of the correlation peak is lower than thepre-determined threshold value, it may be determined that correlationhas been lost.

If it is determined that correlation has not been lost, the method 1600may continue eye-gaze tracking by tracking speckle patterns (1606). Ifit is determined that correlation has been lost, the method 1600 may goback to eye-gaze tracking using the camera (1602). When a new startingeye-gaze position is obtained (1604) by using camera tracking, themethod 1600 may resume eye-gaze tracking by tracking speckle patterns(1606).

As discussed above, eye-gaze tracking using speckle patterns may be moreaccurate than eye-gaze tracking using the camera. In some embodiments,the method 1600 may choose the best camera frames for determiningstarting eye-gaze positions, and perform speckle tracking in betweencamera frames for better accuracy. In some embodiments, the method 1600may perform speckle tracking at a fairly high frame rate, e.g., 10,000fps, and perform camera tracking at a much lower frame rate, e.g., 3fps.

According to another embodiment of the present invention, an initialabsolute eye-gaze position may be obtained by using pre-calibratedspeckle patterns. Eye speckle patterns may be unique to each individual.Before tracking eye-gaze of a user, a calibration procedure may beperformed for the user. In the calibration procedure, a number ofspeckle patterns corresponding to various eye-gaze positions of the usermay be obtained. The various eye-gaze positions may be determined bycamera-based methods or other suitable eye-gaze tracking methods. Thus,a speckle map unique to the user may be obtained and stored in thesystem. The speckle map may include a plurality of pre-calibratedspeckle patterns and their corresponding eye-gaze positions.

FIG. 17 illustrates a simplified flowchart illustrating a method 1700 ofeye-gaze tracking according to an embodiment of the present invention.The method 1700 includes performing eye-gaze tracking using apre-calibrated speckle map for a user (1702), and determining if astarting eye-gaze position has been obtained (1704). As discussed above,the speckle map may include a plurality of pre-calibrated specklepatterns and their corresponding eye-gaze positions. A plurality ofinitial speckle patterns may be obtained by directing a light beam atthe eye and detecting the plurality of initial speckle patterns at adetector positioned adjacent the eye. The light beam may be produced bya coherent light source such as a laser, or by a partially coherentlight source such as light emitting diodes (LEDs), so that specklepatterns may be detected. The plurality of initial speckle patterns maybe compared with the pre-calibrated speckle patterns in the speckle mapto determine an initial eye-gaze position. For example, it may bedetermined that one of the plurality of initial speckle patterns matchesa speckle pattern stored in the speckle map for the user. Then thecorresponding eye-gaze position for the matching speckle pattern may beused as the starting eye-gaze position.

If a starting position has not been obtained, the method 1700 maycontinue eye-gaze tracking using the speckle map (1702). Once a startingposition is obtained, the system may start tracking eye-gaze relative tothe starting eye-gaze position by tracking the speckle patterns (1706).Movement of the eye may be tracked by tracking speckle patterns fromframe to frame using algorithms, such as a phase correlation algorithmof an optical flow method.

The method 1700 further includes determining whether a blinking hasoccurred (1708), and whether correlation has lost as a result of ablinking (1708), similar to the method 1600 discussed above in relationto FIG. 16. If it is determined that correlation has not been lost, themethod 1700 may continue speckle tracking (1706). If it is determinedthat correlation has been lost, the method 1700 may go back to step 1702to perform eye-gaze tracking using the speckle map (1702). When a newstarting position is obtained (1704), the method 1700 may resume speckletracking (1706). In some embodiments, the method 1700 may performspeckle tracking at a fairly high frame rate, e.g., 10,000 fps, andperform tracking using the speckle map at a much lower frame rate, e.g.,3 fps.

FIG. 18 is a simplified flowchart illustrating a method 1800 of trackingmovement of an eye of a user according to an embodiment of the presentinvention. The method 1800 includes directing a light beam at the eye(1802). The light beam may be produced by a coherent light source suchas a laser, or by a partially coherent light source such as lightemitting diodes (LEDs), so that speckle patterns may be detected. Theeye may reflect a portion of the light beam. The portion of the lightbeam reflected by the eye may be diffusely or specular reflected by theeye. The method 1800 further includes detecting a plurality of specklepatterns formed at a detector by the portion of the light beam reflectedby the eye (1804). The plurality of speckle patterns is detected at apredetermined frame rate. The method 1800 further includes trackingmovement of the eye by tracking the plurality of speckle patterns fromframe to frame (1806). In some embodiments, the predetermined frame rateis greater than about 5,000 frames per second and less than about 15,000frames per second. In one embodiment, the predetermined frame rate isabout 10,000 frames per second. In some other embodiments, thepredetermined frame rate is greater than about 50 frames per second andless than about 15,000 frames per second. In some embodiments, trackingthe plurality of speckle patterns is performed using an optical flowalgorithm. In one embodiment, the optical flow algorithm uses a phasecorrelation method.

FIG. 19 is a simplified flowchart illustrating a method 1900 of trackinga gaze of an eye of a user according to an embodiment of the presentinvention. The method 1900 includes recording a first plurality ofimages of the eye using a camera at a first frame rate (1902); anddetermining a starting position of the eye based on the first pluralityof images of the eye (1904). In some embodiments, one of the firstplurality of images may be selected for determining a starting positionof the eye. Based on the selected image, a starting position of the eyemay be determined using a pre-calibrated coordinate system. For example,as illustrated in FIG. 1, the (x, y) coordinates of the center of thepupil may be determined relative to a pre-calibrated origin.

The method 1900 may further include directing a light beam at the eye(1906). The light beam may be produced by a coherent light source suchas a laser, or by a partially coherent light source such as lightemitting diodes (LEDs). The eye may reflect a portion of the light beam.The portion of the light beam reflected by the eye may be diffusely orspecular reflected by the eye. The method 1900 may further includedetecting a first plurality of speckle patterns formed at the detectorby the portion of the light beam reflected by the eye (1908), andtracking movement of the eye relative to the starting position bytracking the first plurality of speckle patterns from frame to frame(1910). In some embodiments, the plurality of speckle patterns aredetected at a second frame rate greater than the first frame rate. Insome embodiments, the first frame rate is less than about 10 fps, andthe second frame rate is greater than about 50 fps and less than about15,000 fps. In one embodiment, the first frame rate is about 3 framesper second, and the second frame rate is about 10,000 frames per second.

In some embodiments, the method 1900 may further include determiningthat the eye has blinked (1920). For example, it may be determinedwhether the eye has blinked based on a tracking quality score asdiscussed above in relation to FIGS. 16 and 17. The method 1900 may goback to step 1902 to determine a new starting position in response todetermining that the eye has blinked. For example, the method 1900 mayinclude determining a new starting position by: recording a secondplurality of images of the eye using the camera, and determining the newstarting position based on the second plurality of images of the eye;and tracking movement of the eye relative to the new starting positionby: detecting a second plurality of speckle patterns formed at thedetector at the second frame rate, and tracking the second plurality ofspeckle patterns from frame to frame. If it is determined that that theeye has not blinked, the method 1900 may continue with steps 1908 and1910

FIG. 20 is a simplified flowchart illustrating a method 2000 of trackinga gaze of an eye of a user according to some embodiments of the presentinvention. The method 2000 may include obtaining a pre-calibratedspeckle map of the eye of the user (2010), determining a startingposition of the eye using the pre-calibrated speckle map (2020), andtracking movement of the eye relative to the starting position (2030),as further described below in relation to FIGS. 21-23.

FIG. 21 is a simplified flowchart illustrating a method 2010 ofobtaining a pre-calibrated speckle map of the eye of the user accordingto an embodiment of the present invention. The method 2010 includesdirecting a first light beam at the eye (2011). The first light beam maybe produced by a coherent light source such as a laser, or by apartially coherent light source such as light emitting diodes (LEDs).The eye may reflect a portion of the first light beam. The method 2010further includes detecting a plurality of first speckle patterns formedat a detector by the portion of the first light beam reflected by theeye (2012). Each of the plurality of first speckle patterns correspondsto a respective first position of the eye. The method 2010 furtherincludes recording a plurality of images of the eye using a camera(2013). Each of the plurality of images of the eye is recorded when arespective first speckle pattern is detected. The method 2010 furtherincludes determining the respective first position of the eyecorresponding to each respective first speckle pattern based on therespective image of the eye (2014), and storing the plurality of firstspeckle patterns and the corresponding first positions of the eye(2015).

FIG. 22 is a simplified flowchart illustrating a method 2020 ofdetermining a starting position of the eye using the pre-calibratedspeckle map according to an embodiment of the present invention. Themethod 2020 includes directing a second light beam at the eye (2021).The second light beam may be produced by a coherent light source such asa laser, or by a partially coherent light source such as light emittingdiodes (LEDs). The eye may reflect a portion of the second light beam.The method 2020 further includes detecting a plurality of second specklepatterns formed at the detector by the portion of the second light beamreflected by the eye (2022), and determining the starting position ofthe eye based on comparisons between the plurality of second specklepatterns and the stored plurality of first speckle patterns and thestored corresponding first positions of the eye (2023).

FIG. 23 is a simplified flowchart illustrating a method 2030 of trackingmovement of the eye relative to the starting position according to anembodiment of the present invention. The method 2030 includes directinga third light beam at the eye (2031). The third light beam may beproduced by a coherent light source such as a laser, or by a partiallycoherent light source such as light emitting diodes (LEDs). The eye mayreflect a portion of the third light beam. The method 2030 furtherincludes detecting a plurality of third speckle patterns formed at thedetector by the portion of the third light beam reflected by the eye(2032). The plurality of third speckle patterns being detected at apredetermined frame rate. The method 2030 further includes trackingmovement of the eye relative to the starting position by tracking thethird plurality of speckle patterns from frame to frame.

As discussed above, eye speckle patterns may be unique to eachindividual. Thus, eye speckle patterns can be used for useridentification. FIG. 24 is a simplified flowchart illustrating a method2400 of identification of a user according to an embodiment of thepresent invention. The method 2400 includes directing a first light beamat the first eye (2402). The first light beam may be produced by acoherent light source such as a laser, or by a partially coherent lightsource such as light emitting diodes (LEDs). The first eye may reflect aportion of the first light beam. The method 2400 further includesdetecting a plurality of first speckle patterns formed at a detector bythe portion of the first light beam reflected by the first eye (2404).Each of the plurality of first speckle patterns corresponds to arespective first position of the first eye. The method 2400 furtherincludes recording a plurality of images of the first eye using a camera(2406). Each of the plurality of images of the first eye is recordedwhen a respective first speckle pattern is detected. The method 2400further includes determining the respective first position of the firsteye corresponding to each respective first speckle pattern based on therespective image of the first eye (2408), and storing the plurality offirst speckle patterns and the corresponding first positions of thefirst eye (2410). The method 2400 further includes directing a secondlight beam at an eye of a user (2412). The second light beam may beproduced by a coherent light source such as a laser, or by a partiallycoherent light source such as light emitting diodes (LEDs). The eye mayreflect a portion of the second light beam. The method 2400 furtherincludes detecting a plurality of second speckle patterns formed at thedetector by the portion of the second light beam reflected by the eye(2414). Each of the plurality of second speckle patterns corresponds toa respective second position of the eye. The method 2400 furtherincludes determining that the user is the first user by comparing theplurality of second speckle patterns with the plurality of first specklepatterns (2416).

It should be appreciated that the specific steps illustrated in each ofFIGS. 16-24 provide a particular method according to an embodiment ofthe present invention. Other sequences of steps may also be performedaccording to alternative embodiments. For example, alternativeembodiments of the present invention may perform the steps outlinedabove in a different order. Moreover, the individual steps illustratedin FIGS. 16-24 may include multiple sub-steps that may be performed invarious sequences as appropriate to the individual step. Furthermore,additional steps may be added or removed depending on the particularapplications. One of ordinary skill in the art would recognize manyvariations, modifications, and alternatives.

It is also understood that the examples and embodiments described hereinare for illustrative purposes only and that various modifications orchanges in light thereof will be suggested to persons skilled in the artand are to be included within the spirit and purview of this applicationand scope of the appended claims.

What is claimed is:
 1. A method of tracking a gaze of an eye of a user,the method comprising: obtaining a pre-calibrated speckle map of the eyeof the user by: directing a first light beam at the eye, the eyereflecting a portion of the first light beam; detecting a plurality offirst speckle patterns formed at a detector by the portion of the firstlight beam reflected by the eye, each of the plurality of first specklepatterns corresponding to a respective first position of the eye;recording a plurality of images of the eye using a camera, each of theplurality of images of the eye being recorded when a respective firstspeckle pattern is detected; determining the respective first positionof the eye corresponding to each respective first speckle pattern basedon the respective image of the eye; and storing the plurality of firstspeckle patterns and the corresponding first positions of the eye;determining a starting position of the eye using the pre-calibratedspeckle map by: directing a second light beam at the eye, the eyereflecting a portion of the second light beam; detecting a plurality ofsecond speckle patterns formed at the detector by the portion of thesecond light beam reflected by the eye; and determining the startingposition of the eye based on comparisons between the plurality of secondspeckle patterns and the stored plurality of first speckle patterns andthe stored corresponding first positions of the eye; and trackingmovement of the eye relative to the starting position by: directing athird light beam at the eye, the eye reflecting a portion of the thirdlight beam; detecting a plurality of third speckle patterns formed atthe detector by the portion of the third light beam reflected by theeye, the plurality of third speckle patterns being detected at apredetermined frame rate; and tracking movement of the eye relative tothe starting position by tracking the third plurality of specklepatterns from frame to frame.
 2. The method of claim 1 wherein each ofthe portion of the first light beam reflected by the eye, the portion ofthe second light beam reflected by the eye, and the portion of the thirdlight beam reflected by the eye is diffusely reflected or specularreflected by the eye.
 3. The method of claim 1 wherein tracking thethird plurality of speckle patterns is performed using an optical flowalgorithm.
 4. The method of claim 3 wherein the optical flow algorithmcomprises a phase correlation algorithm.
 5. The method of claim 1wherein the predetermined frame rate is greater than about 50 frames persecond and less than about 15,000 frames per second.
 6. The method ofclaim 1 further comprising: determining that the eye has blinked;determining a new starting position of the eye by: directing a fourthlight beam at the eye, the eye reflecting a portion of the fourth lightbeam; detecting a plurality of fourth speckle patterns formed at thedetector by the portion of the fourth light beam reflected by the eye;and determining the new starting position of the eye based oncomparisons between the plurality of fourth speckle patterns and thestored plurality of first speckle patterns and the stored correspondingfirst positions of the eye; and tracking movement of the eye relative tothe new starting position by: directing a fifth light beam at the eye,the eye reflecting a portion of the fifth light beam; detecting aplurality of fifth speckle patterns formed at the detector by theportion of the fifth light beam reflected by the eye, the plurality offifth speckle patterns being detected at the predetermined frame rate;and tracking movement of the eye relative to the new starting positionby tracking the fifth plurality of speckle patterns from frame to frame.7. The method of claim 1 wherein each of the first light beam, thesecond light beam, and the third light beam comprises infraredradiation.